{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "926bada6"
      },
      "source": [
        "Copyright 2025 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "a110dfce"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pkOtOq0jDE0c"
      },
      "source": [
        "<table align=\"left\">\n",
        "  <td>\n",
        "      <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/gemma-cookbook/blob/main/Demos/Emoji-Gemma-on-Web/resources/Fine_tune_Gemma_3_270M_for_emoji_generation.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e624ec07"
      },
      "source": [
        "# Fine-tune Gemma 3 270M for emoji generation\n",
        "\n",
        "This notebook fine-tunes Gemma for the task of translating text into emoji using Quantized Low-Rank Adaptation (QLoRA) through the Hugging Face Transformer Reinfocement Learning ([TRL](https://huggingface.co/docs/trl/en/index)) library to help reduce memory usage and speed up the fine-tuning process.\n",
        "\n",
        "When training [Gemma 3 270M](https://huggingface.co/google/gemma-3-270m) on a Google Colab T4 GPU accelerator, this process can take as little as 10 minutes end-to-end. Run each code snippet to:\n",
        "\n",
        "1. Set up the Colab environment\n",
        "2. Prepare a dataset for fine-tuning\n",
        "3. Load and test the base Gemma 3 270M model\n",
        "4. Fine-tune the model\n",
        "5. Test, evaluate, and save the model for further use\n",
        "\n",
        "## Set up development environment\n",
        "\n",
        "The first step is to install the necessary libraries using the `pip` package installer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BEK9IfKBqQaA"
      },
      "outputs": [],
      "source": [
        "%pip install torch tensorboard emoji\n",
        "%pip install -U transformers trl datasets accelerate evaluate sentencepiece bitsandbytes protobuf==3.20.3"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TTuW1LPfLXi9"
      },
      "source": [
        "You may have the restart your session (runtime) to use newly installed libraries."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ef3d54b"
      },
      "source": [
        "##Enable Hugging Face permissions\n",
        "To use Gemma models, you'll need to accept the model usage license and create an Access Token:\n",
        "\n",
        "1. **Accept license** on the [model page](http://huggingface.co/google/gemma-3-270m-it).\n",
        "\n",
        "2. **Get a valid [Access Token](https://huggingface.co/settings/tokens) with 'Write' access (very important!)**\n",
        "\n",
        "3. Create a new Colab secret in the left toolbar. Specify `HF_TOKEN` as the 'Name', add your unique token as the 'Value', and toggle 'Notebook access' on."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b6d79c93"
      },
      "outputs": [],
      "source": [
        "from google.colab import userdata\n",
        "from huggingface_hub import login\n",
        "\n",
        "# Login into Hugging Face Hub\n",
        "hf_token = userdata.get('HF_TOKEN')\n",
        "login(hf_token)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "42c60525"
      },
      "source": [
        "## Load the dataset\n",
        "\n",
        "Hugging Face hosts a large collection of datasets for training and evaluating models. If you're not using a custom dataset, you can load a [premade dataset](https://huggingface.co/datasets/kr15t3n/g-emoji) containing examples of text and corresponding emoji translations.\n",
        "\n",
        "**If you'd like to use your own custom dataset, skip to the next step.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bc3BYl72pWhp"
      },
      "outputs": [],
      "source": [
        "import emoji\n",
        "from datasets import load_dataset\n",
        "\n",
        "# Use the first 2000 samples for efficient training.\n",
        "general_dataset_path = load_dataset(\"kr15t3n/text2emoji\", encoding=\"utf-8\", split=\"train\")\n",
        "\n",
        "# Clean dataset to only use examples where 'emoji' field contains only emoji characters\n",
        "def is_only_emoji(sample):\n",
        "  emoji_string = sample['emoji']\n",
        "  if not emoji_string:\n",
        "    return False\n",
        "  return all(emoji.is_emoji(char) for char in emoji_string)\n",
        "dataset = general_dataset_path.filter(is_only_emoji)\n",
        "\n",
        "print(f\"\\nHere's the 10th example from the dataset: {dataset[10]}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kc1rhhdFGwNu"
      },
      "source": [
        "### Upload a custom dataset\n",
        "**If you've already loaded a dataset, skip this step.**\n",
        "\n",
        "\n",
        "You can customize Gemma 3 270M to use specific emoji by creating a spreadsheet containing your text-to-emoji dataset structured as key-value pairs. If you want to encourage memorization of specific emoji, we recommend providing 10-20 examples of that emoji with different text variations.\n",
        "\n",
        "Use a [premade dataset](https://github.com/google-gemini/gemma-cookbook/blob/main/Demos/Emoji-Gemma-on-Web/resources/Emoji%20Translation%20Dataset%20-%20Dataset.csv)  as a template to create your own dataset, then upload it the Files folder in the left toolbar. Get its path by right-clicking the file and pointing to it in `custom_dataset_path`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g0HGgZj_3f7g"
      },
      "outputs": [],
      "source": [
        "import emoji\n",
        "from datasets import load_dataset\n",
        "from transformers import AutoTokenizer\n",
        "\n",
        "# Point to your uploaded dataset\n",
        "custom_dataset_path = \"/content/Emoji Translation Dataset - Dataset.csv\"      #@param {type:\"string\"}\n",
        "dataset = load_dataset(\"csv\", data_files=custom_dataset_path, encoding=\"utf-8\", split=\"train\")\n",
        "\n",
        "# Clean dataset to only use examples where 'emoji' field contains only emoji characters\n",
        "def is_only_emoji(sample):\n",
        "  emoji_string = sample['emoji']\n",
        "  if not emoji_string:\n",
        "    return False\n",
        "  return emoji.purely_emoji(emoji_string)\n",
        "dataset = dataset.filter(is_only_emoji)\n",
        "\n",
        "print(f\"\\nHere's the 10th example from your dataset: {dataset[10]}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c0eb2e06"
      },
      "source": [
        "## Load the model\n",
        "\n",
        "You can access [Gemma 3 270M](https://huggingface.co/google/gemma-3-270m-it) from Hugging Face Hub by accepting the license terms. The instruction-tuned version of the model has already been trained on how to follow directions and with fine-tuning, you'll now adapt it to a new task."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "18069ed2"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from transformers import AutoTokenizer, AutoModelForCausalLM\n",
        "\n",
        "gemma_model = \"google/gemma-3-270m-it\"\n",
        "base_model = AutoModelForCausalLM.from_pretrained(gemma_model, device_map=\"auto\", attn_implementation=\"eager\", dtype=torch.bfloat16)\n",
        "tokenizer = AutoTokenizer.from_pretrained(gemma_model)\n",
        "\n",
        "print(f\"Device: {base_model.device}\")\n",
        "print(f\"DType: {base_model.dtype}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7hI4twbrz0xj"
      },
      "source": [
        "Device should print as `cuda` if you're using a GPU runtime. **If you haven't aleady, use a free T4 GPU runtime in your Colab for faster fine-tuning.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PafivP8u1Gv9"
      },
      "source": [
        "### Format the training dataset\n",
        "Now that you've loaded your data, format the training dataset into conversational roles, including the text input and emoji output, plus a system prompt that contains the direction to the model. This helps the model learn how to interpret the 'text' and 'emoji' columns from your dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VWz32s5h074E"
      },
      "outputs": [],
      "source": [
        "from transformers import AutoTokenizer\n",
        "\n",
        "def translate(sample):\n",
        "  return {\n",
        "      \"messages\": [\n",
        "          {\"role\": \"system\", \"content\": \"Translate this text to emoji: \"},\n",
        "          {\"role\": \"user\", \"content\": f\"{sample['text']}\"},\n",
        "          {\"role\": \"assistant\", \"content\": f\"{sample['emoji']}\"}\n",
        "      ]\n",
        "  }\n",
        "\n",
        "training_dataset = dataset.map(translate, remove_columns=dataset.features.keys())\n",
        "training_dataset_splits = training_dataset.train_test_split(test_size=0.1, shuffle=True)\n",
        "\n",
        "print(\"\\nHere's the 40th example from the formatted training dataset:\")\n",
        "print(training_dataset[40])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M3w3b9-O4fDz"
      },
      "source": [
        "### Recommended: Test the base model\n",
        "\n",
        "Let's first check how the base model's ability to respond to the instruction \"Translate this text to emoji\"\n",
        "\n",
        "Try testing it a few times."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "u8L0_INJyUok"
      },
      "outputs": [],
      "source": [
        "from transformers import pipeline\n",
        "from random import randint\n",
        "import re\n",
        "\n",
        "# Create a transformers inference pipeline\n",
        "pipe = pipeline(\"text-generation\", model=gemma_model, tokenizer=tokenizer)\n",
        "\n",
        "# Select a random sample from the test dataset\n",
        "rand_idx = randint(0, len(training_dataset_splits[\"test\"]) - 1)\n",
        "test_sample = training_dataset_splits[\"test\"][rand_idx]\n",
        "\n",
        "# Handle messages\n",
        "all_messages = test_sample['messages']\n",
        "user_message_content = next((msg['content'].strip() for msg in all_messages if msg['role'] == 'user'), \"Not Found\")\n",
        "dataset_emoji = next((msg['content'].strip() for msg in all_messages if msg['role'] == 'assistant'), \"Not Found\")\n",
        "prompt_messages = [\n",
        "    {\"role\": \"system\", \"content\": \"Translate this text to emoji: \"},\n",
        "    {\"role\": \"user\", \"content\": user_message_content}\n",
        "]\n",
        "\n",
        "# Apply the chat template. This will format the messages correctly for the model.\n",
        "prompt = tokenizer.apply_chat_template(prompt_messages, tokenize=False, add_generation_prompt=True)\n",
        "\n",
        "# Generate the output\n",
        "output = pipe(prompt, max_new_tokens=64)\n",
        "model_output_only = output[0]['generated_text'][len(prompt):].strip()\n",
        "\n",
        "print(f\"\\nDataset text: {user_message_content}\")\n",
        "print(f\"\\nDataset emoji: {dataset_emoji}\")\n",
        "print(f\"\\nModel generated output: {model_output_only}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ph26HDJgua3W"
      },
      "source": [
        "The base model output may not meet your expectations—and that's okay!\n",
        "\n",
        "Gemma 3 270M was designed for task specialization, which means it can improve performance for specific tasks when trained with representative examples. Let's fine-tune the model for more reliable outputs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bbd9fc1b"
      },
      "source": [
        "## Fine-tune the model\n",
        "\n",
        "Hugging Face [TRL](https://huggingface.co/docs/trl/index) provides tools for training and fine-tuning LLMs using memory-efficient techniques like QLoRA (Quantized Low-Rank Adaptation) to train adapters on top of a frozen quantized version of the model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-BJFoOdL0y8w"
      },
      "source": [
        "### Configure the tuning job\n",
        "Define the training configuration for the Gemma 3 base model:\n",
        "\n",
        "1. `BitsandBytesConfig` to quantize the model for memory efficiency\n",
        "2. `LoraConfig` for parameter-efficient fine-tuning\n",
        "2. `SFTConfig` for supervised fine-tuning"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qiIj1ADc-exw"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n",
        "from peft import LoraConfig\n",
        "from trl import SFTConfig\n",
        "\n",
        "adapter_path = \"/content/myemoji-gemma-adapters\"      # Where to save your LoRA adapters\n",
        "tokenizer = AutoTokenizer.from_pretrained(gemma_model)\n",
        "\n",
        "bnb_config = BitsAndBytesConfig(\n",
        "    load_in_4bit=True,\n",
        "    bnb_4bit_quant_type=\"nf4\",\n",
        "    bnb_4bit_compute_dtype=torch.bfloat16\n",
        ")\n",
        "\n",
        "lora_config = LoraConfig(\n",
        "    r=16,\n",
        "    lora_alpha=32,\n",
        "    target_modules=\"all-linear\",                      # Target all linear layers\n",
        "    lora_dropout=0.05,                                # Increase to 0.1 to induce overfitting\n",
        "    bias=\"none\",\n",
        "    task_type=\"CAUSAL_LM\",\n",
        "    modules_to_save=[\"lm_head\", \"embed_tokens\"]       # Save the lm_head and embed_tokens as you train the special tokens\n",
        ")\n",
        "\n",
        "args = SFTConfig(\n",
        "    output_dir=adapter_path,                          # Directory to save adapters\n",
        "    num_train_epochs=3,                               # Number of training epochs\n",
        "    per_device_train_batch_size=4,                    # Batch size per device during training\n",
        "    logging_strategy=\"epoch\",                         # Log every epoch\n",
        "    eval_strategy=\"epoch\",                            # Evaluate loss metrics every epoch\n",
        "    save_strategy=\"epoch\",                            # Save checkpoint every epoch\n",
        "    learning_rate=5e-5,                               # Learning rate,\n",
        "    lr_scheduler_type=\"constant\",                     # Use constant learning rate scheduler\n",
        "    max_length=256,                                   # Max sequence length for model and packing of the dataset\n",
        "    gradient_checkpointing=False,                     # Use gradient checkpointing to save memory\n",
        "    packing=False,                                    # Groups multiple samples in the dataset into a single sequence\n",
        "    optim=\"adamw_torch_fused\",                        # Use fused adamw optimizer\n",
        "    report_to=\"tensorboard\",                          # Report metrics to tensorboard\n",
        "    weight_decay=0.01,                                # Added weight decay for regularization\n",
        ")\n",
        "\n",
        "base_model = AutoModelForCausalLM.from_pretrained(gemma_model, quantization_config=bnb_config, device_map=\"auto\", attn_implementation='eager')\n",
        "base_model.config.pad_token_id = tokenizer.pad_token_id\n",
        "\n",
        "print(\"Training configured\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dd88e798"
      },
      "source": [
        "### Start training\n",
        "\n",
        "`SFTTrainer` tokenizes the datasets and trains the base model using the hyperparameters from the previous step.\n",
        "\n",
        "The training time varies based on a range of factors, such as the size of your dataset or number of epochs. Using a T4 GPU, this takes about 10 minutes for 1000 training examples. If training is going slowly, check that you're using a T4 GPU in Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WqacJNeU9v7b"
      },
      "outputs": [],
      "source": [
        "from trl import SFTConfig, SFTTrainer\n",
        "\n",
        "# Set training and evaluation datasets\n",
        "train_dataset = training_dataset_splits['train']\n",
        "eval_dataset = training_dataset_splits['test']\n",
        "\n",
        "# Train and save the LoRA adapters\n",
        "trainer = SFTTrainer(\n",
        "    model=base_model,\n",
        "    args=args,\n",
        "    train_dataset=train_dataset,\n",
        "    eval_dataset=eval_dataset,\n",
        "    peft_config=lora_config,\n",
        ")\n",
        "trainer.train()\n",
        "trainer.save_model(adapter_path)\n",
        "\n",
        "print(f\"LoRA adapters saved to {adapter_path}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dDvGlb5xO34z"
      },
      "source": [
        "The LoRA adapters for each training checkpoint (epoch) will be saved in your temporary Colab session storage. Now, you can evaluate the training and validation loss metrics to choose which adapters to merge with the model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xll8zZ3_u8Mt"
      },
      "source": [
        "### Plot training results\n",
        "To evaluate the model, you can plot the training and validation losses using Matplotlib to visualize these metrics over training steps or epochs. This helps monitor the training process and make informed decisions about hyperparameters or early stopping."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vPN-DTopaUIy"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Access the log history\n",
        "log_history = trainer.state.log_history\n",
        "\n",
        "# Extract training / validation loss\n",
        "train_losses = [log[\"loss\"] for log in log_history if \"loss\" in log]\n",
        "epoch_train = [log[\"epoch\"] for log in log_history if \"loss\" in log]\n",
        "eval_losses = [log[\"eval_loss\"] for log in log_history if \"eval_loss\" in log]\n",
        "epoch_eval = [log[\"epoch\"] for log in log_history if \"eval_loss\" in log]\n",
        "\n",
        "# Plot the training loss\n",
        "plt.plot(epoch_train, train_losses, label=\"Training Loss\")\n",
        "plt.plot(epoch_eval, eval_losses, label=\"Validation Loss\")\n",
        "plt.xlabel(\"Epoch\")\n",
        "plt.ylabel(\"Loss\")\n",
        "plt.title(\"Training and Validation Loss per Epoch\")\n",
        "plt.legend()\n",
        "plt.grid(True)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vyIwS-orvWzd"
      },
      "source": [
        "Training loss measures the error on the data the model was trained on. Validation loss measures the error on a separate dataset the model has not seen before. Monitoring both helps detect overfitting (when the model performs well on training data but poorly on unseen data).\n",
        "\n",
        "- validation loss >> training loss: **overfitting**\n",
        "- validation loss > training loss: **some overfitting**\n",
        "- validation loss < training loss: **some underfitting**\n",
        "- validation loss << training loss: **underfitting**\n",
        "\n",
        "If your task requires memorization of specific examples, or specific emoji to be generated for a given text, overfitting can be beneficial."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ILgEZvZ71Edz"
      },
      "source": [
        "### Merge the adapters\n",
        "\n",
        "Once trained you can merge the LoRA adapters with the model. You can choose which adapters to merge by specifying the training checkpoint folder, otherwise it will default to the last epoch.\n",
        "* For better task generalization, choose the most underfit checkpoint (validation loss < training loss)\n",
        "* For better memorization of specific examples, choose the most overfit (checkpoint > training loss)  \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "A7e5BQ9U06Q2"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from transformers import AutoTokenizer, AutoModelForCausalLM\n",
        "from peft import PeftModel\n",
        "\n",
        "adapter_path = \"/content/myemoji-gemma-adapters/\"                 # Choose which adapters to merge, otherwise defaults to latest\n",
        "merged_model_path = \"/content/myemoji-gemma-merged/\"              # Location of merged model directory\n",
        "\n",
        "# Load base model and tokenizer\n",
        "base_model = AutoModelForCausalLM.from_pretrained(gemma_model, device_map=\"auto\")\n",
        "tokenizer = AutoTokenizer.from_pretrained(adapter_path)\n",
        "\n",
        "# Load and merge the PEFT adapters onto the base model\n",
        "model = PeftModel.from_pretrained(base_model, adapter_path)\n",
        "model = model.merge_and_unload()\n",
        "\n",
        "# Save the merged model and its tokenizer\n",
        "model.save_pretrained(merged_model_path)\n",
        "tokenizer.save_pretrained(merged_model_path)\n",
        "\n",
        "print(f\"Model merged and saved to {merged_model_path}. Final model vocabulary size: {model.config.vocab_size}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bf86e31d"
      },
      "source": [
        "### Test the fine-tuned model\n",
        "\n",
        "Let's compare your fine-tuned model performance against the base model! Test a few inputs by updating `text_to_translate`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "28R3pRN_hai7"
      },
      "outputs": [],
      "source": [
        "from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline\n",
        "\n",
        "# Create Transformers inference pipeline\n",
        "merged_model = AutoModelForCausalLM.from_pretrained(merged_model_path, device_map=\"auto\")\n",
        "tokenizer = AutoTokenizer.from_pretrained(merged_model_path)\n",
        "pipe = pipeline(\"text-generation\", model=merged_model, tokenizer=tokenizer)\n",
        "pipe_base = pipeline(\"text-generation\", model=gemma_model, device_map=\"auto\")\n",
        "\n",
        "# Test a prompt\n",
        "text_to_translate = \"let's go to the beach\"  #@param {type:\"string\"}\n",
        "inference_messages = [\n",
        "    {\"role\": \"system\", \"content\": \"Translate this text to emoji: \"},\n",
        "    {\"role\": \"user\", \"content\": text_to_translate}\n",
        "]\n",
        "prompt = tokenizer.apply_chat_template(inference_messages, tokenize=False, add_generation_prompt=True)\n",
        "output = pipe(prompt, max_new_tokens=128)\n",
        "output_base = pipe_base(prompt, max_new_tokens=128)\n",
        "model_output = output[0]['generated_text'][len(prompt):].strip()\n",
        "model_output_base = output_base[0]['generated_text'][len(prompt):].strip()\n",
        "\n",
        "print(f\"\\nFine-tuned model output: {model_output}\")\n",
        "\n",
        "print(f\"\\nBase model output: {model_output_base}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "86qPcFbHH_kh"
      },
      "source": [
        "Does the model output the emoji you'd expect?\n",
        "\n",
        "If you're not getting the results you want, you can try [using different hyperparameters](#scrollTo=-BJFoOdL0y8w) to train the model, or updating your training dataset to contain more representative examples.\n",
        "\n",
        "Once you're happy with the results, you can save your model to Hugging Face Hub."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H12D9g4X_peV"
      },
      "source": [
        "## Save your model and upload to Hugging Face Hub\n",
        "**You now have a customized Gemma 3 270M model! 🎉**\n",
        "\n",
        "Upload it to a repository on Hugging Face Hub so you easily share your model or access it later."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VbeyDcpwi4IB"
      },
      "outputs": [],
      "source": [
        "from huggingface_hub import ModelCard, ModelCardData, whoami\n",
        "\n",
        "#@markdown Name your model\n",
        "model_name = \"myemoji\"                            #@param {type:\"string\"}\n",
        "\n",
        "username = whoami()['name']\n",
        "hf_repo_id = f\"{username}/{model_name}-gemma-3-270m-it\"\n",
        "\n",
        "repo_url = model.push_to_hub(hf_repo_id, create_repo=True, commit_message=\"Upload model\")\n",
        "tokenizer.push_to_hub(hf_repo_id)\n",
        "\n",
        "card_content = f\"\"\"\n",
        "---\n",
        "base_model: {gemma_model}\n",
        "tags:\n",
        "- text-generation\n",
        "- emoji\n",
        "- gemma\n",
        "---\n",
        "A fine-tuned model based on `{gemma_model}`.\"\"\"\n",
        "card = ModelCard(card_content)\n",
        "card.push_to_hub(hf_repo_id)\n",
        "\n",
        "print(f\"Uploaded to {repo_url}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6f8ff452"
      },
      "source": [
        "## Summary and next steps\n",
        "\n",
        "This notebook covered how to efficiently fine-tune Gemma 3 270M for emoji generation. Continue on to the conversion and quantization steps to get it ready for on-device deployment. You can follow the steps to either:\n",
        "\n",
        "1.  [Convert for use with MediaPipe LLM Inference API](https://colab.research.google.com/github/google-gemini/gemma-cookbook/blob/main/Demos/Emoji-Gemma-on-Web/resources/Convert_Gemma_3_270M_to_LiteRT_for_MediaPipe_LLM_Inference_API.ipynb)\n",
        "2.  [Convert for use with Transformers.js via ONNX Runtime](https://colab.research.google.com/github/google-gemini/gemma-cookbook/blob/main/Demos/Emoji-Gemma-on-Web/resources/Convert_Gemma_3_270M_to_ONNX.ipynb)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "Fine_tune_Gemma_3_270M_for_emoji_generation.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
